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Abstract 


Department  of  Defense  (DoD)  spending  on  services  has  been  trending 
upwards  for  over  a  decade  and,  as  of  201 1 ,  it  accounted  for  56%  of  total  contract 
spending.  The  increased  reliance  on  services  contractors  has  prompted  the 
Government  Accountability  Office  (GAO)  to  look  more  closely  at  the  acquisition  and 
contract  management  process.  In  this  research  we  address  the  following  questions: 
(1)  How  do  different  stakeholders  define  successful  services  contracts  within  the 
Navy?  (2)  How  do  different  stakeholders  measure  services  contracts  within  the 
Navy?  and  (3)  how  should  Navy  services  contracts  be  defined  and  measured?  We 
conducted  a  survey  of  168  key  stakeholders.  We  discovered  that  when  defining  and 
measuring  the  success  of  a  service  contract  all  stakeholders  tend  to  utilize  outcome- 
related  factors  over  process-oriented  factors.  We  believe  this  is  because  outcomes 
tend  to  drive  perceptions  of  success  more  than  processes  and  are  more  easily 
quantifiable.  Metrics  used  to  measure  success  are  typically  related  to  cost, 
schedule,  and  performance.  Based  on  these  findings,  we  provide  recommendations 
on  establishing  better  internal  control  measures,  putting  in  place  an  operational  audit 
process,  and  creating  a  standardized  reporting  process. 

Keywords:  Services  Acquisition,  Services  Contracts,  Success  of  Services 
Contracts 
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I. 


Introduction 


The  service  sector  represents  the  largest  and  the  fastest  growing  segment  of 
the  economies  of  the  U.S.  and  other  developed  countries.  This  growth  of  services  in 
the  overall  economy  is  also  mirrored  by  the  growth  of  services  acquisition  in  the 
Department  of  Defense  (DoD).  For  example,  the  DoD  obligations  on  contracts  have 
more  than  doubled  between  fiscal  years  2001  and  2008  to  over  $387  billion,  with 
over  $200  billion  spent  just  for  services  in  2008  (Government  Accountability  Office 
[GAO],  2009).  In  conjunction  with  this  increase  in  defense  procurement  is  the 
reduction  of  the  defense  acquisition  workforce.  The  size  of  the  federal  workforce 
decreased  from  2.25  million  in  1990  to  1 .78  million  in  2000  (GAO,  2002).  The 
combination  of  the  increasing  defense  procurement  workload  and  the  decreasing 
size  of  the  government  workforce,  along  with  the  complexities  of  an  arcane  and 
convoluted  government  contracting  process,  have  created  the  perfect  storm — an 
environment  in  which  complying  with  government  contracting  policies  and  adopting 
contract  management  best  practices  has  not  always  been  feasible  (Rendon,  2010). 
Between  2001  and  2009,  the  Government  Accountability  Office  (GAO)  issued  16 
reports  related  to  trends,  challenges,  and  deficiencies  in  defense  contracting. 

During  this  same  time  frame,  the  DoD  Inspector  General  (DoDIG)  issued  142  reports 
on  deficiencies  in  the  DoD  acquisition  and  contract  administration  processes.  These 
reports  have  identified  poor  contract  planning,  contract  administration,  and 
contractor  oversight  as  just  some  of  the  critically  deficient  areas  in  DoD  contract 
management.  Because  of  these  deficiencies,  the  GAO  has  identified  contract 
management  as  a  “high  risk”  area  for  the  federal  government  since  1990  and 
continues  to  identify  it  as  high  risk  (GAO,  2013). 

As  the  DoD’s  services  acquisition  continues  to  increase  in  scope  and  dollars, 
the  agency  must  give  greater  attention  to  proper  acquisition  planning,  adequate 
requirements  definition,  sufficient  price  evaluation,  and  proper  contractor  oversight 
(GAO,  2002).  In  fact,  as  stressed  in  a  recent  memorandum  for  acquisition 


- 1  - 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


professionals  by  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and 
Logistics  (USD[AT&L],  2010),  improving  the  efficiency  of  the  acquisition  of  products 
and  services  is  of  utmost  importance  to  the  DoD.  In  some  ways,  the  issues  affecting 
services  acquisition  are  similar  to  those  affecting  the  acquisition  of  physical  supplies 
and  weapon  systems.  However,  the  unique  characteristics  of  services  and  the 
increasing  importance  of  services  acquisition  offer  a  significant  opportunity  for 
conducting  research  in  the  management  of  services  acquisition  in  the  Department  of 
Defense. 

A.  Research  Questions 

This  research  project  undertakes  a  focused,  in-depth  study  of  the  services 
acquisition  so  as  to  understand  how  success  of  service  acquisition  contracts  is  being 
defined  and  measured  in  the  Navy.  The  contract  management  process  is  performed 
with  inputs  from  the  different  functional  areas,  using  a  cross-functional  team  or 
integrated  project  team  (IPT)  structure.  The  team  members  represent  their 
functional  areas  such  as  program  management,  contracting,  financial,  logistics,  and 
quality  assurance.  Each  of  these  project  team  members  represents  the 
stakeholders,  and  their  different  goals  and  objectives,  as  discussed  in  the  previous 
section.  For  example,  project  managers  are  generally  focused  on  meeting  the  cost, 
schedule,  and  performance  objectives  of  the  contract.  Yet  these  objectives  could 
possibly  be  in  conflict  with  the  contracting  officer’s  objectives  of  protecting  the 
government’s  interest  and  ensuring  that  the  contract  is  in  compliance  with  statutory 
requirements  (for  example,  providing  for  full  and  open  competition)  and  public  policy 
objectives  (for  example,  supporting  small  and  minority-owned  businesses).  In 
addition,  the  contract’s  end  user  may  have  contract  objectives  that  are  different  from 
those  of  the  project  manager  and  contracting  officer.  Based  on  these  potentially 
incongruent  objectives,  each  functional  area  may  consider  a  contract’s  success 
differently  than  the  other  functional  areas.  Hence,  the  first  research  question  we 
investigated  was  as  follows:  How  do  different  stakeholders  define  successful 
services  contracts  within  the  Navy?  To  develop  a  clear  understanding  of  current 
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services  acquisition  practices,  we  also  investigated  the  second  research  question: 
How  do  different  stakeholders  measure  services  contracts  within  the  Navy? 
Investigating  the  previous  two  questions  helped  us  develop  recommendations 
regarding  the  third  and  final  research  question:  How  should  the  service  contract’s 
success  be  measured?  The  next  section  provides  a  literature  review  of  some  of  the 
management  theories  informing  service  supply  chain  management,  as  well  as  some 
of  our  previous  research  on  DoD  services  acquisition. 
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Literature  Review 


The  academic  research  in  the  management  of  services  acquisition  is  founded 
on  several  economic  and  management  theories  including  agency  theory 
(Eisenhardt,  1989),  transaction  cost  economics  (Williamson,  1979),  contractual 
theory  (Luo,  2002),  service  operations  and  supply  management  (Fitzsimmons  & 
Fitzsimmons,  2006),  and  stakeholder  theory  (Freeman,  1984;  Cleland  ,1986;  El- 
Gohary,  Osman,  &  El-Diraby,  2006).  In  addition  to  providing  a  brief  review  of  the 
agency  theory  and  the  stakeholder  theory,  we  also  provide  a  summary  of  research 
projects  carried  out  by  the  authors  in  the  area  of  services  supply  chain. 

A.  Agency  Theory 

Agency  theory  is  reflected  in  a  contract  between  the  government  and  a 
contractor  forming  a  principal-agent  relationship.  The  principal  (government) 
contracts  with  the  agent  (contractor)  to  perform  some  level  of  effort,  such  as 
developing  or  manufacturing  a  product  or  providing  a  service.  In  this  relationship,  the 
government’s  objectives  include  obtaining  the  product  or  service  at  the  right  quality, 
right  quantity,  right  source,  right  time,  and  right  price  (Lee  &  Dobler,  1971).  The 
federal  government  also  has  the  additional  objective  of  ensuring  the  product  or 
service  is  procured  in  accordance  with  public  policy  and  statutory  requirements 
(Federal  Acquisition  Regulation  [FAR],  2012).  Contractors,  on  the  other  hand, 
pursue  the  objectives  of  earning  a  profit,  ensuring  company  growth,  maintaining  or 
increasing  market  share,  and  improving  cash  flow,  just  to  name  a  few.  Because  of 
the  different  and  conflicting  objectives  between  the  principal  and  agent,  each  party  is 
motivated  and  incentivized  to  behave  in  a  certain  manner.  This  behavior  includes 
either  withholding  or  sharing  information.  In  principal-agent  relationships  that  involve 
higher  levels  of  uncertainty,  which  result  in  higher  risk  (such  as  developing  an 
advanced  technology  weapon  system),  the  information  available  to  the  government 
and  contractor  is  typically  asymmetrical.  Thus,  agency  theory  is  concerned  with  the 
conflicting  goals  between  the  principal  and  agent  in  obtaining  their  respective 
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objectives  and  is  focused  on  mechanisms  related  to  obtaining  information  (for 
example,  about  the  marketplace,  the  supply  or  service,  or  the  contractor),  selecting 
the  agent  (to  counter  the  problem  of  adverse  selection),  and  monitoring  the  agent’s 
performance  (to  counter  the  effects  of  moral  hazard).  Thus,  decisions  about  how 
contracts  are  planned  (for  example,  competitive  or  sole  source),  structured  (fixed 
price  or  cost  reimbursement,  with  or  without  incentives),  awarded  (based  on  lowest 
priced,  technically  acceptable  offer,  or  the  highest  technically  rated  offer),  and 
administered  (centralized  or  decentralized,  level  and  type  of  surveillance,  and  use  of 
project  teams,  etc.)  have  their  basis  in  agency  theory  and  the  principal-agent 
problem.  Agency  theory  can  also  be  applied  to  project  management,  specifically  in 
the  management  of  services  acquisition  projects  (Moe,  1984).  In  services  acquisition 
projects,  the  same  principal-agent  model  exists.  The  principal  in  this  perspective  is 
the  project  manager,  and  the  agents  are  the  other  members  of  the  project  team,  for 
example,  the  contracting  officer  and  financial  manager.  The  project  manager  is 
faced  with  the  problem  of  ensuring  the  agents,  in  this  context  the  members  of  the 
project  team,  will  choose  to  pursue  the  principal’s  best  interests. 

B.  Stakeholder  Theory 

Stakeholder  theory  can  also  be  used  to  analyze  DoD  services  acquisition.  In 
stakeholder  theory,  the  purpose  of  the  organization  (for  example,  a  DoD  agency)  is 
to  generate  and  dispense  some  form  of  wealth  to  various  stakeholders  (individuals 
that  have  a  stake  or  interest  in  the  agency),  and,  in  order  to  achieve  that  purpose,  all 
of  the  stakeholders  cooperate  (Freeman,  1984).  Thus,  stakeholder  theory,  as 
applied  to  DoD  services  acquisition,  includes  all  of  the  major  stakeholders 
concerned  with  the  services  acquisition  project  including  the  program/project 
manager  (PM),  contracting  officer  (CO),  contracting  officer  representative  (COR), 
financial  manager,  and  the  customers  who  use  the  service.  The  program/project 
manager  is  responsible  for  ensuring  that  the  desired  services  acquisition  results  (in 
terms  of  cost,  schedule,  and  performance)  are  achieved.  The  contracting  officer  is 
an  agent  of  the  government  with  the  authority  to  enter  into,  administer,  and/or 
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terminate  contracts  and  make  related  determinations  and  findings  (FAR,  2012).  The 
contracting  officer  is  responsible  for  making  sure  that  contracts  are  planned, 
executed,  and  closed  out  in  accordance  with  agency  regulations  and  statutory 
requirements  (Rendon  &  Snider,  2008).  Contracting  officers  support  the  PM  by 
providing  contracting  support  to  help  achieve  the  contract  objectives  of  cost, 
schedule,  and  performance.  Due  to  the  technical  and  complex  nature  of  the  DoD’s 
acquired  services,  the  contracting  officer  representative  (COR)  is  an  integral 
member  of  the  acquisition  team.  The  COR  is  the  individual  providing  technical 
expertise  of  the  service  being  procured  and  is  formally  appointed  in  writing  by  the 
contracting  officer.  The  COR  is  involved  in  the  entire  acquisition  and  procurement 
process  and  assists  the  contracting  officer  in  developing  the  technical  documents 
associated  with  the  procurement.  Thus,  the  COR  supports  the  PM  and  CO  by 
monitoring  all  of  the  day-to-day  technical  aspects  of  the  contract  to  include 
inspecting  and  accepting  the  services  provided  by  the  contractor.  The  finance 
manager  is  another  stakeholder  in  DoD  services  acquisition,  serving  as  the  fiscal 
and  budgetary  advisor  to  the  program  manager.  The  finance  manager  is 
responsible  for  ensuring  compliance  with  the  statutory  requirements  of  fiscal  law 
(e.g.,  that  proper  authorization  is  granted  for  expending  funds,  the  contract 
obligations  occur  during  the  time  limits  prescribed  by  appropriation,  and  adequate 
funding  is  available;  Rendon  &  Snider,  2008).  The  customer’s  role  in  the  services 
acquisition  process  is  to  have  sufficient  knowledge  of  the  requirement  so  that  it  can 
be  clearly  communicated  to  industry  through  the  contracting  process.  The  customer 
plays  a  vital  role  in  the  services  acquisition  process  because  this  is  the  stakeholder 
that  determines  how  well  the  service  requirement  is  documented  in  the  contracting 
process.  Although  the  PM,  CO,  COR,  financial  manager,  and  customer  are  all 
important  stakeholders  of  a  service  acquisition,  they  each  have  differing  roles  and 
responsibilities,  and  their  definitions  of  success  may  vary  and  may  even  be  in 
conflict  with  each  other. 
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C.  Services  Supply  Chain  Management 

We  have  addressed  the  need  for  research  in  this  increasingly  important  area 
of  services  acquisition  by  undertaking  six  sponsored  research  projects  over  the  past 
six  years.  The  first  two  research  projects  (Apte,  Ferrer,  Lewis,  &  Rendon,  2006; 

Apte  &  Rendon,  2007)  were  exploratory  in  nature,  aimed  at  understanding  the  types 
of  services  being  acquired,  the  associated  rates  of  growth  in  services  acquisition, 
and  the  major  challenges  and  opportunities  present  in  the  service  supply  chain  . 

The  next  two  research  projects  were  survey-based  empirical  studies  aimed  at 
developing  a  high-level  understanding  of  how  services  acquisition  is  currently  being 
managed  at  a  wide  range  of  Army,  Navy,  and  Air  Force  installations  (Apte,  Apte,  & 
Rendon,  2008;  Apte,  Apte,  &  Rendon,  2009).  The  analysis  of  survey  data  indicated 
that  the  current  state  of  services  acquisition  management  suffers  from  several 
deficiencies  including  deficit  billet  and  manning  levels  (which  are  further  aggravated 
by  insufficient  training  and  the  inexperience  of  acquisition  personnel),  and  the  lack  of 
strong  project-team  and  life-cycle  approaches.  Our  research  (Apte,  Apte,  &  Rendon, 
2010)  also  analyzed  and  compared  the  results  of  the  primary  data  collected  in  two 
previous  empirical  studies  involving  Army,  Navy,  and  Air  Force  contracting 
organizations  so  as  to  develop  a  more  thorough  and  comprehensive  understanding 
of  how  services  acquisition  is  being  managed  within  individual  military  Services. 

As  a  result  of  these  research  projects  dealing  with  the  service  supply  chain  in 
the  DoD,  we  have  developed  a  comprehensive,  high-level  understanding  of  services 
acquisition  in  the  DoD,  have  identified  several  specific  deficiencies,  and  have 
proposed  a  number  of  concrete  recommendations  for  performance  improvement. 

Based  on  the  foundation  of  the  previously  mentioned  management  theories, 
conclusions  of  the  GAO  and  DoDIG  reports  (Seifert  &  Ermoshkin,  2010),  and 
findings  of  our  own  sponsored  research  projects  on  the  topic,  we  believe  that  the 
success  of  service  acquisition  contracts  is  significantly  influenced  by  four  broadly 
defined  factors:  (1)  the  type  and  quantity  of  services  being  outsourced  and  the 
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associated  amount  of  acquisition-related  workload;  (2)  the  characteristics  of 
contracts  being  awarded;  (3)  the  capacity  available  to  carry  out  the  contracting, 
project  management,  and  surveillance  work;  and  (4)  various  management  practices 
such  as  use  of  project  team  or  life-cycle  approaches  and  so  forth.  A  conceptual 
model  indicating  the  interrelationship  among  these  factors  is  shown  in  Figure  1 . 


Acquired  Services 

Contract 

-  Service  Type 

Characteristics 

-  Contractual  Amounts 

- 1 

*  -  Level  of  Competition 

-  Acquisition  Workload 

-  Contract  Type 

-  Incentives 

Capacity 

Mgmt.  Practices 

-  No.  of  Billets 

-  Project  Mgmt 

-  Fill  Rate 

-  Life  Cycle 

-  Training 

-  Leadership 

-  Surveillance 

Etc. 

Success  of  Services 
Contracts 

-  Efficiency 

-  Effectiveness 

-  Compliance 


Figure  1.  Drivers  of  Acquisition  Practices  and  Success  of 

Service  Contracts 

As  shown  in  the  conceptual  diagram  of  Figure  1 ,  the  contract  characteristics 
are  affected  by  the  type  of  service  being  acquired,  while  the  management  practices 
being  used  are  influenced  by  the  services  being  acquired,  the  contract 
characteristics,  and,  more  importantly,  the  capacity  available  to  perform  the 
acquisition  work.  The  success  of  services  contracts,  in  turn,  is  affected  by  the 
previously  mentioned  four  drivers.  Underlying  Figure  1  is  the  fundamental  question 
motivating  our  in-depth  research:  what  drives  the  success  of  services  contracts? 

This  fundamental  question  is,  of  course,  critically  important,  and  yet  it  is  also  not  one 
that  can  be  answered  easily  or  quickly.  We  believe  that,  generally,  in  the  case  of 
questions  related  to  complex  systems,  it  is  preferable  to  break  down  the  overall 
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system  in  smaller  parts,  gain  an  understanding  of  the  functioning  of  each  part,  and 
then  put  all  the  pieces  together  to  better  understand  the  overall  system  and  answer 
the  fundamental  question. 

The  objectives  of  this  research  project  are  to  (1 )  understand  how  the  success 
of  services  contracts  is  being  defined  by  different  stakeholders,  (2)  identify  how  the 
success  of  services  contracts  is  currently  being  measured,  and  (3)  develop  specific 
recommendations  on  how  the  success  of  services  contracts  should  be  measured. 
We  address  our  research  methodology  in  the  next  section. 
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Research  Methodology 


With  the  assistance  of  our  MBA  thesis  students  (Hagan,  Spede,  &  Sutton, 
2012),  we  developed  and  deployed  a  data  collection  survey  instrument  to  collect 
empirical  data  for  answering  our  research  questions.  The  survey  was  deployed  to 
the  various  stakeholders  at  the  participating  commands.  We  then  analyzed  the  data 
using  descriptive  statistics  to  provide  recommendations  and  conclusions. 

We  developed  and  deployed  a  web-based  survey  using  the  SurveyMonkey 
website.  The  survey  instrument  included  both  demographic  questions  and  core 
questions  related  to  defining  and  measuring  successful  services  contracts.  The  core 
questions  were  designed  to  establish  the  importance  of  different  factors  when 
defining  and  measuring  the  success  of  services  contracts.  These  core  questions 
were  related  to  the  contracting  process,  as  well  as  to  different  outcomes  such  as 
cost,  schedule,  and  performance  (Hagan,  Spede,  &  Sutton,  2012). 

In  terms  of  defining  successful  contracts,  the  core  questions  asked 
participants  to  rank  various  definitions  relating  to  the  four  metrics  (process,  cost, 
schedule,  and  performance)  in  order  of  most  important  (1 )  to  least  important  (5).  We 
also  asked  participants  to  rate  definition  statements  relating  to  process,  cost, 
schedule,  and  performance.  These  questions  use  a  Likert  scale  asking  level  of 
agreement,  importance,  and  amount  of  time  devoted  by  the  participants.  The  Likert 
scale  had  a  range  of  1  to  5,  with  1  representing  a  negative  response  and  5 
representing  a  positive  response  (Hagan,  Spede,  &  Sutton,  2012). 

In  terms  of  measuring  successful  contracts,  the  core  questions  asked 
participants  to  rank  various  measurements  relating  to  the  four  metrics  in  order  of 
most  important  (1 )  to  least  important  (5).  The  last  question  in  the  section  asks 
participants  to  rate  on  a  Likert  scale  how  often  the  organization  conducts  certain 
actions  that  pertain  to  the  measurement  of  success  concerning  process,  schedule, 
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cost,  and  performance.  Figure  2  reflects  our  survey  question  approach  (Hagan, 
Spede,  &  Sutton,  2012). 


Figure  2.  Diagram  of  Survey  Questions 

The  survey  was  deployed  to  the  major  stakeholders  (PMs,  COs,  and  CORs) 
at  the  following  major  contracting  commands:  Fleet  Logistics  Center  (FLC) 
Philadelphia,  FLC  Jacksonville,  FLC  Norfolk,  FLC  Puget  Sound,  FLC  San  Diego, 
Naval  Sea  Systems  Command  (NAVSEA),  Military  Sealift  Command  (MSC),  and 
Space  and  Naval  Warfare  Systems  Command  (SPA WAR;  Hagan,  Spede,  &  Sutton, 
2012). 
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IV. 


Survey  Results  and  Analysis 


In  this  section,  we  present  the  results  of  the  survey  and  discuss  its  major 
findings.  As  mentioned  previously,  the  primary  objective  of  this  research  is  to 
empirically  examine  how  the  success  of  a  service  contract  is  being  defined  and 
measured  by  different  stakeholders.  We  designed  a  survey  containing  19  questions 
and  distributed  them  to  the  major  stakeholders  in  the  services  acquisition  process  to 
receive  their  responses.  The  survey  was  deployed  at  eight  Navy  installations 
identified  previously.  We  distributed  the  survey  to  a  total  of  843  respondents 
responsible  for  various  acquisition-related  functions.  Specifically,  we  surveyed  the 
following  stakeholders:  program  manager/project  officer,  contract  officer/contract 
specialist,  contracting  officer  representative,  requirements  manager,  financial 
manager,  contractor,  and  customer.  The  survey  questions  included  both  Likert-type 
as  well  as  ranking-type  questions.  The  Likert-type  questions  were  used  to  assess 
favorable  or  unfavorable  responses,  while  the  ranking-type  questions  were  used  to 
assess  the  most  important  responses.  When  we  examine  the  ranking  questions  in 
this  section,  the  term  “most  important”  refers  to  the  number  of  factors  that  received 
the  highest  rankings  of  1  or  2.  We  believe  that  this  is  the  best  way  to  capture  and 
succinctly  represent  the  participants’  responses.  For  example,  a  COR  may  feel  that 
the  outcome-related  factors  are  extremely  important  and,  therefore,  should  be  given 
the  highest  ranking  of  1  every  time.  However,  the  COR  may  also  believe  that  the 
process-related  factors  are  very  important,  too,  and  hence  may  assign  the  next 
highest  rank  of  2  to  those  factors.  Hence,  we  believe  that  the  percent  of  respondents 
giving  a  rank  of  1  or  2  to  a  factor  is  the  most  effective  way  to  capture  and  represent 
the  importance  of  that  factor  while  analyzing  the  data  on  ranking  of  factors. 

The  survey  response  rates  we  experienced  for  different  categories  of 
stakeholders  are  shown  in  Table  1 .  Unfortunately,  we  received  only  a  small  number 
of  responses  from  requirements  managers,  financial  managers,  contractors,  and 
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customers.  Hence,  their  responses  are  not  incorporated  in  this  report  for  analysis 
purposes.  These  respondents  are  combined  under  the  “other”  category  in  Table  1 . 


Table  1.  Survey  Response  Rate 


STAKEHOLDER 

#  SURVEYS 
DEPLOYED 

#  SURVEYS 
ANSWERED 

RESPONSE 

RATE 

PROGRAM  MANAGER/PROJECT  OFFICER 

94 

15 

16% 

CONTRACTING  OFFICER 

REPRESENTATIVE 

104 

27 

26% 

CONTRACTING  OFFICER/  CONTRACT 
SPECIALIST 

280 

126 

45% 

AGGREGATE  DATA  (PM,  COR,  PCO) 

478 

168 

35% 

OTHER 

365 

10 

2.7% 

TOTAL 

843 

178 

21% 

We  present  the  survey  results  and  analysis  in  three  sub-sections:  the  first 
sub-section  presents  the  aggregate  data,  the  second  sub-section  presents  the 
stakeholder-level  data,  and  the  third  sub-section  presents  the  service-type  data.  A 
set  of  tables  summarizing  the  survey  data  is  presented  in  Appendix  A. 

A.  Survey  Results:  Aggregate  Survey  Data 

1.  Defining  the  Success  of  a  Service  Contract 

In  taking  a  high-level  view  of  our  survey  findings,  we  did  not  differentiate 
between  functional  roles,  DAWIA  levels  of  certification,  type  of  service  being 
acquired,  contract  type,  or  the  organization.  However,  we  did  separate  our  findings 
under  the  broad  categories  of  process  and  outcome.  Outcome  results  included  the 
questions  associated  with  cost,  schedule,  and  performance.  As  shown  in  Table  1, 
collectively,  there  were  168  responses  from  PMs,  CORs,  or  PCOs.  The  Likert  scale 
responses  were  assigned  a  value  of  1  through  5,  with  the  higher  value  representing 
a  more  favorable  response  to  a  statement.  A  summary  of  aggregate  data  about 
defining  and  measuring  the  success  of  a  service  contract  is  presented  in  Tables  2 
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and  3  of  Appendix  A.  We  examined  the  mean  of  responses  to  each  set  of  Likert 
scale-type  questions.  We  found  that  when  defining  the  success  of  a  services 
contract,  outcomes  are  considered  slightly  more  important  than  processes.  The 
overall  mean  of  responses  related  to  outcomes  was  4.08,  while  process  responses 
resulted  in  a  mean  of  3.97.  Our  findings  are  displayed  graphically  in  Figure  3. 

We  then  separated  our  findings  further  within  the  broad  category  of  outcomes 
into  the  narrower  categories  of  cost,  schedule,  and  performance.  Performance- 
related  questions  resulted  in  the  highest  mean  of  4.29,  while  cost-related  questions 
produced  a  mean  of  4.03,  and  schedule-related  questions  produced  a  mean  of  3.93. 

One  hundred  and  sixty-eight  respondents  were  asked  to  rank  different  factors 
related  to  defining  the  success  of  a  service  contract.  These  questions  also  dealt 
with  different  aspects  of  processes  and  outcomes.  Of  the  168  respondents,  40%  felt 
that  process-related  factors  were  the  most  important.  Sixty  percent  felt  that 
outcome-related  factors  were  the  most  important.  The  distribution  of  highest  ranked 
responses  is  displayed  in  Figure  4. 

Breaking  down  the  outcome-related  factors  further,  15%  of  respondents  felt 
that  cost-related  factors  were  the  most  important,  19%  felt  that  schedule-related 
factors  were  most  important,  and  26%  felt  that  performance-related  factors  were 
most  important. 


NAVAL  POSTGRADUATE  SCHOOL 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 


-  15  - 


Means  of  Aggregate  Stakeholder 
Definitions  of  Success  (n=168) 
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Figure  3.  Means  of  Aggregate  Stakeholder  Definitions  of  Success 


Aggregate  Stakeholder  Ranking 
Definitions  of  Success 


Figure  4.  Aggregate  Stakeholder  Ranking  of  Definitions  of  Success 
2.  Measuring  the  Success  of  a  Service  Contract 

Our  survey  also  requested  that  participants  rate  on  the  Likert  scale  the 
various  degrees  of  importance,  and  the  extent  to  which  they  agreed  or  disagreed 
with,  various  factors  when  considering  how  they  measure  the  success  of  a  service 
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contract.  Again,  these  factors  related  to  either  processes  or  outcomes.  The  overall 
Likert  scale  mean  with  relation  to  processes  was  2.48,  and  the  outcomes  displayed 
an  overall  mean  of  3.71 .  Clearly  outcomes  are  deemed  more  important  by  our 
participants  as  a  whole.  Our  findings  are  displayed  graphically  in  Figure  5. 

If  we  look  at  the  distinct  factors  within  outcome  of  cost,  schedule,  and 
performance,  the  overall  Likert  means  were  3.96,  3.84,  and  3.30,  respectively. 

One  hundred  and  sixty-eight  respondents  were  asked  to  rank  different  factors 
related  to  measuring  the  success  of  a  service  contract.  Of  the  168  respondents, 
46%  felt  that  process-related  factors  were  the  most  important.  Fifty-four  percent  felt 
that  outcome-related  factors  were  the  most  important.  The  distribution  of  highest 
ranked  responses  is  displayed  in  Figure  6. 

Breaking  down  the  outcome-related  factors  further,  19%  of  respondents  felt 
that  cost-related  factors  were  the  most  important,  12%  felt  that  schedule-related 
factors  were  most  important,  and  23%  felt  that  performance-related  factors  were 
most  important. 
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Figure  6.  Aggregate  Stakeholder  Ranking  of  Measurements 

of  Success 

3.  Analysis  of  Aggregate  Survey  Data 

The  findings  from  the  analysis  of  aggregate  survey  data  show  that  when 
asked  to  respond  on  a  Likert  scale,  different  stakeholders  find  all  aspects  of 
processes  and  outcomes  important  when  defining  the  success  of  a  service  contract. 
The  means  of  the  responses  we  collected  are  very  close,  and  it  does  not  seem  that, 
as  a  whole,  our  population  favors  process  or  outcome  when  defining  success. 
Perhaps  this  is  due  to  the  nature  of  Likert  scale  questions.  When  asked  if  something 
such  as  cost  overruns,  major  milestones,  or  a  lack  of  protests  is  important,  all 
stakeholders  will  invariably  say  yes.  That  is  why  the  overall  mean  of  all  responses, 
for  both  outcomes  and  processes,  is  fairly  high  at  4.03.  When  forced  to  rank,  the 
responses  differ  and  outcome-related  responses  received  a  high  rank  of  1  or  2  60% 
of  the  time.  This  is  because  outcomes  such  as  keeping  on  schedule  and  budget 
adherence  are  easy  to  understand  and  define.  Process-related  factors  such  as 
administration  and  communication  are  relatively  harder  to  quantify. 

The  findings  also  demonstrate  that  when  measuring  the  success  of  a  service 
contract,  all  stakeholders  tend  to  focus  on  outcomes  and  do  not  take  into 
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consideration  the  processes;  this  was  true  for  both  Likert  scale  responses  and 
ranking  responses.  This  is  very  evident  in  the  Likert  scale  responses,  where  none  of 
the  process-related  factors  showed  a  mean  of  3  or  more.  When  forced  to  rank  the 
different  factors  with  respect  to  measuring  success,  the  results  were  similar  to 
defining  success,  with  56%  of  “most  important”  responses  falling  under  the 
outcomes  category. 

In  general  our  findings  from  the  “other”  category  mirrored  our  aggregate 
results.  While  there  were  only  10  responses,  all  felt  that  outcomes  were  the  most 
important  factor  when  defining  and  measuring  the  success  of  a  service  contract.  We 
found  that  our  stakeholders  in  this  category  rated  and  ranked  processes  extremely 
low  in  both  defining  and  measuring  the  success  of  a  service  contract.  This  is 
because  these  stakeholders  are  not  terribly  burdened  by  administration  and  other 
process-related  factors  so  they  feel  that  these  factors  are  not  important.  For 
example,  a  contractor  or  end  user  does  not  necessarily  conduct  market  research  or 
choose  the  appropriate  contract  type.  However,  they  are  very  concerned  with 
staying  within  cost,  keeping  up  with  schedule,  and  maintaining  a  high  level  of 
performance. 


B.  Survey  Results:  Stakeholder-Level  Data 

As  a  starting  point  in  examining  how  different  stakeholders  define  and 
measure  the  success  of  a  service  contract,  we  performed  a  statistical  analysis  of  the 
data  to  determine  if  there  were  significant  differences  between  the  ratings  on  the 
Likert  scale  across  the  major  stakeholders.  We  first  performed  an  F-test  for  sample 
variances  to  determine  the  appropriate  f-test  to  perform.  In  all  instances,  we  found 
that  there  was  an  equal  variance  among  stakeholders.  The  only  statistically 
significant  difference  was  between  the  CORs  and  CO/specialists  when  measuring 
success.  This  could  be  due  to  the  fact  that  CORs  view  communication  and  other 
processes  as  a  key  factor  when  measuring  the  success  of  a  service  contract.  The 
COR  is  also  likely  to  view  a  protest  as  a  serious  issue  when  measuring  success 
because  it  results  in  a  delay  of  execution  and  CORs  cannot  perform  their  duties. 
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Otherwise,  there  was  no  statistically  significant  difference  between  any  other  of  the 
stakeholders  on  the  Likert  scale.  Given  these  results,  we  do  not  plan  to  present 
charts  presenting  and  comparing  how  different  stakeholders  define  and  measure 
success  of  a  service  contract.  However,  a  summary  of  how  different  stakeholders 
define  and  measure  the  success  of  a  service  contract  is  presented  in  Tables  4  and  5 
of  Appendix  A.  We  discuss  in  the  next  section  the  results  of  the  analysis  of 
stakeholder-level  data. 

1.  Analysis  of  Stakeholder-Level  Data 

Consistent  with  the  abovementioned  results  of  statistical  analysis,  we  found 
that  PMs,  CORs,  COs,  and  contract  specialists  all  agree  that  outcome  is  slightly 
more  important  than  processes,  based  on  participants’  ratings  of  separate  factors  on 
a  Likert  scale.  Each  functional  role  rated  outcome  slightly  over  4.00,  while  rating 
processes  just  below  4.00.  The  mean  of  the  functional  roles  combined  was  3.94  for 
processes,  and  4.1 1  for  outcomes.  Within  outcome,  performance-related  factors 
received  the  highest  average  rating,  while  schedule-related  factors  received  the 
lowest  average  rating.  All  functional  roles  showed  an  upward  trend  from  schedule, 
to  cost,  to  performance.  A  comparison  of  our  Likert  scale  findings  for  defining 
success  across  functional  roles  is  displayed  graphically  in  Figure  7. 

When  stakeholders  were  asked  to  rank  different  factors  concerning  their 
definition  of  success,  we  found  that  there  was  clear  agreement  that  outcomes  are 
more  important  than  processes.  There  was,  however,  some  disagreement  within  the 
outcome  factors  of  cost,  schedule,  and  performance.  CORs  felt  that  cost  was  the 
most  important  factor,  while  PMs,  COs,  and  specialists  placed  performance  at  the 
top  of  their  rankings.  Examined  collectively,  the  major  stakeholders  provided  168 
responses  when  ranking  their  definition  of  the  success  of  a  service  contract.  Sixty 
percent  of  respondents  felt  that  outcome-related  factors  were  most  important,  while 
40%  felt  that  process-related  factors  were  the  most  important  when  defining 
success.  The  distribution  of  highest  ranked  responses  is  displayed  in  Figure  8. 
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Figure  7.  Definitions  of  Success  Across  Major  Stakeholders 
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Figure  8.  Major  Stakeholder  Ranking  of  Definitions  of  Success 

According  to  the  survey  data,  stakeholders  also  tend  to  measure  success  in 
almost  the  same  way.  When  asked  to  rate  different  factors  on  the  Likert  scale 
related  to  stakeholders’  measures  of  success,  all  respondents  agreed  that  outcomes 
far  outweigh  processes.  When  looking  at  the  mean  across  stakeholders,  processes 
received  a  rating  of  2.56,  while  outcomes  received  a  rating  of  3.78.  Within  outcome- 
related  factors,  stakeholders  showed  an  upward  trend  from  performance,  to 
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schedule,  to  cost.  A  comparison  of  our  findings  for  defining  success  on  the  Likert 
scale  across  functional  roles  is  displayed  graphically  in  Figure  9. 

Our  ranking  data  shows  that,  again,  major  stakeholders  prefer  outcome- 
related  factors  when  measuring  the  success  of  service  contracts.  When  examined 
in  aggregate,  the  major  stakeholders  provided  168  responses  to  our  ranking 
questions.  Of  these  responses,  43%  of  respondents  felt  process  factors  were  most 
important,  while  57%  were  in  favor  of  factors  related  to  outcomes.  The  distribution 
of  highest  ranked  responses  is  displayed  in  Figure  10. 
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Figure  9.  Measurement  of  Success  Across  Major  Stakeholders 
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Figure  10.  Major  Stakeholder  Ranking  of  Measurements 

of  Success 

The  Likert  scale  responses  for  definitions  of  success  were,  again,  relatively 
high,  and  this  was  due  to  the  reason  explained  earlier.  It  is  interesting  that  in  both 
defining  and  measuring  success,  CORs  ranked  cost  highest  out  of  the  three 
stakeholders. 

Another  interesting  result  is  that  COs  tended  to  place  nearly  equal  importance 
on  process  and  outcomes  when  forced  to  rank  factors  concerning  measuring 
success.  This  is  probably  due  to  the  administrative  nature  of  the  COs’  role.  For 
example,  their  functional  role  has  to  deal  with  modifications,  COR  reports,  and 
exercising  options.  The  other  functional  roles  of  PMs  and  CORs  are  not  overly 
concerned  with  processes  and  are  focused  on  the  requirement  and  outcomes.  The 
data  reflect  this  fact. 

It  is  interesting  to  note  that  every  demographic  consistently  rated  processes 
significantly  higher  on  the  Likert  scale  when  defining  success  versus  measuring 
success.  We  feel  that  this  is  because  stakeholders  view  measures  as  a  tangible 
entity  associated  with  post-award  functions.  Measures  such  as  cost,  schedule,  and 
performance  are  fairly  straightforward  in  as  much  as  either  a  goal  is  met  or  it  is  not. 
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Processes  such  as  communication  flow  and  overall  management  are  more  obscure 
and  subjective.  The  stakeholders  rated  processes  higher  for  defining  success 
because  they  are  closely  associated  with  mainly  pre-award  functions.  Processes 
such  as  choosing  the  correct  contract  type  and  appropriately  evaluating  the  proposal 
are  crucial  for  success.  Because  these  are  pre-award  activities,  it  is  easier  to  define 
success  rather  than  measure  it. 

C.  Survey  Results:  Type  of  Service 

While  examining  our  data,  we  found  that  the  majority  of  responses  were  from 
participants  who  procured  two  types  of  services.  Of  the  168  total  responses,  the 
equipment-related  service  portfolio  grouping  accounted  for  72  responses  and  the 
knowledge-based  service  portfolio  grouping  for  77  responses.  The  equipment- 
related  service  portfolio  grouping  includes  the  procurement  of  maintenance,  repair 
and  overhaul,  equipment  modification,  installation,  and  quality  control.  The 
knowledge-based  services  portfolio  grouping  is  composed  of  professional  and 
administrative  services,  engineering  management,  program  management,  logistics 
management,  and  education  and  training.  Given  that  the  majority  of  responses  were 
from  these  two  groups,  we  examined  and  compared  how  stakeholders  involved  with 
equipment-related  services  and  knowledge-based  services,  respectively,  define  and 
measure  success  of  services  contracts.  We  only  differentiated  by  type  of  service 
and  made  no  other  demographic  distinctions  for  mean  and  ranking  results.  A 
summary  of  survey  data  regarding  how  stakeholders  acquiring  these  two  services 
define  and  measure  the  success  of  a  service  contract  is  presented  in  Tables  6  and  7 
of  Appendix  A. 

An  analysis  of  responses  showed  that  40%  of  knowledge-based  service 
participants  were  involved  with  cost  reimbursement-type  contracts  as  compared  to 
only  6%  for  equipment-related  services.  The  higher  percentage  of  cost 
reimbursement-type  contracts  for  knowledge-based  service  participants  is  most 
likely  due  to  the  increased  challenges  and  uncertainties  in  defining  requirements 
associated  with  these  types  of  services.  Equipment-related  services  are  generally 
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more  concrete  in  terms  of  requirements  definition  and  would  be  more  suitable  for  a 
fixed-price  contractual  instrument. 

1.  Defining  the  Success  of  a  Service  Contract 

In  response  to  our  questions  asking  participants  to  classify  different  factors 
related  to  defining  the  success  of  a  service  contract,  we  received  149  responses 
from  participants  who  work  on  knowledge-based  and  equipment-related  services. 
When  differentiating  between  types  of  service,  we  found  that  equipment-related 
service  participants  rated  both  processes  and  outcomes  higher  on  all  Likert  scale 
questions  than  knowledge-based  service  participants.  Equipment-related 
participants  displayed  a  mean  rating  on  process  and  outcome  factors  of  4.05  and 
4.10,  respectively.  Knowledge-based  participants  displayed  a  mean  rating  on 
process  and  outcome  factors  of  3.92  and  4.02,  respectively.  This  shows  that 
participants  segregated  by  type  of  service  may  rate  processes  and  outcomes  slightly 
differently;  however,  they  both  indicated  that  outcomes  are  slightly  more  important 
than  processes  for  defining  success.  Our  results  are  displayed  graphically  by  type 
of  service  in  Figure  1 1 . 

The  respondents  were  also  asked  to  rank  definitions  of  success  concerning  a 
service  contract.  The  questions  asked  dealt  with  different  aspects  of  defining 
success — some  were  related  to  process  while  others  were  related  to  outcome.  Of 
the  72  respondents  involved  with  procurement  of  equipment-related  services,  42% 
believe  that  process  factors  are  most  important,  while  58%  felt  that  outcomes  more 
accurately  define  the  success  of  a  service  contract.  Breaking  down  the  outcome- 
related  factors  further,  15%  felt  that  cost-related  factors  were  the  most  important, 
18%  felt  that  schedule-related  factors  were  most  important,  and  25%  felt  that 
performance-related  factors  were  most  important. 

Seventy-seven  respondents  involved  with  knowledge-based  services 
provided  responses  when  asked  to  rank  definitions  of  success  concerning  a  service 
contract.  Of  the  77  respondents,  39%  believed  that  process  factors  were  most 
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important,  while  61%  felt  that  outcomes  more  accurately  defined  the  success  of  a 
service  contract.  The  results  show  that  equipment-related  service  participants  rated 
processes  higher  and  outcomes  lower  than  knowledge-based  participants,  but  they 
both  indicated  that  outcomes  are  more  important  than  processes  for  defining 
success,  based  on  their  responses  to  ranking  questions.  The  distribution  of  highest 
ranked  responses  is  displayed  in  Figure  12.  Breaking  down  the  outcome-related 
factors  further,  1 6%  felt  that  cost-related  factors  were  the  most  important,  1 8%  felt 
that  schedule-related  factors  were  most  important,  and  28%  felt  that  performance- 
related  factors  were  most  important. 
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Figure  1 1 .  Definitions  of  Success  by  Service  Type 
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Figure  12.  Ranking  of  Definitions  of  Success  by  Service  Type 
2.  Measuring  the  Success  of  a  Service  Contract 

In  measuring  the  success  for  different  types  of  services,  the  equipment- 
related  service  participants  rated  processes  much  higher  on  Likert  scale  questions 
than  knowledge-based  service  participants.  Equipment-related  service  participants 
also  rated  outcomes  as  being  of  lower  importance  than  did  knowledge-based  service 
participants.  Equipment-related  participants  displayed  a  mean  rating  on  process 
and  outcome  factors  of  2.68  and  3.76,  respectively.  Knowledge-based  participants 
displayed  a  mean  rating  on  process  and  outcome  factors  of  2.29  and  3.82, 
respectively.  The  results  show  that  equipment-related  service  participants  rated 
processes  higher  and  outcomes  lower  than  knowledge-based  participants,  but  they 
both  indicated  that  outcomes  are  more  important  than  processes  for  measuring 
success,  based  on  their  responses  to  Likert  scale  questions.  Our  results  are 
displayed  graphically  by  type  of  service  in  Figure  13. 


Of  the  72  respondents  involved  with  equipment-related  services,  process- 
related  factors  were  ranked  most  important  44%  of  the  time,  while  outcome-related 
factors  were  ranked  as  most  important  56%  of  the  time.  Breaking  down  the 
outcome-related  factors  further,  21%  of  respondents  felt  that  cost-related  factors 
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were  the  most  important,  12%  felt  that  schedule-related  factors  were  most  important, 
and  21%  felt  that  performance-related  factors  were  most  important. 

Forty-eight  percent  of  the  77  respondents  associated  with  knowledge-based 
services  felt  that  process-related  factors  were  the  most  important  when  measuring 
success.  Fifty-two  percent  of  respondents  felt  that  outcome-related  factors  were  the 
most  important.  The  results  show  that  equipment-related  service  participants  rated 
processes  lower  and  outcomes  higher  than  did  knowledge-based  participants,  but 
they  both  indicated  that  outcomes  are  more  important  than  processes  for  measuring 
success,  based  on  their  responses  to  our  ranking  questions.  The  distribution  of 
highest  ranked  responses  is  displayed  in  Figure  14.  Breaking  down  the  outcome- 
related  factors  further,  16%  of  respondents  felt  that  cost-related  factors  were  the 
most  important,  1 1%  felt  that  schedule-related  factors  were  most  important,  and 
25%  felt  that  performance-related  factors  were  most  important. 
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Figure  13.  Measurement  of  Success  by  Service  Type 
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Figure  14.  Ranking  of  Measurement  of  Success  by  Service  Type 

3.  Analysis  of  Service-Type  Data 

The  findings  based  on  type  of  service  showed  no  substantial  deviation  for 
defining  the  success  of  a  services  contract.  Both  equipment-related  and  knowledge- 
based  groups  rated  outcomes  slightly  higher  than  processes,  based  on  their 
responses  to  the  Likert  scale  questions.  Ranking  questions  for  definitions  of 
success  showed  outcomes  as  more  important  than  processes  for  both  groups  and 
corroborated  the  Likert  scale  mean  findings.  Both  Likert  scale  and  ranking  question 
results  show  performance  to  be  the  most  important  component  of  outcomes.  For 
example,  performance  had  means  of  4.28  and  4.24,  compared  to  3.97  and  3.82  for 
schedule.  In  their  responses  to  the  ranking  questions,  25-28%  of  respondents  in 
both  groups  ranked  performance-related  factors  highest,  while  18%  ranked 
schedule-related  factors  highest.  Perhaps  ranking  results,  due  to  the  limitation  of 
choices,  provide  the  most  precise  definitions  of  success.  When  participants  were 
forced  to  rank,  they  emphasized  the  importance  of  outcomes  (cost,  schedule,  and 
performance)  over  processes  more  definitively  when  defining  characteristics  of 
successful  service  contracts. 
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The  findings  for  measuring  success  also  showed  no  substantial  deviation 
based  on  type  of  service.  Both  equipment-related  and  knowledge-based  groups 
rated  outcomes  significantly  higher  than  processes  on  the  Likert  scale  questions. 
Ranking  questions  for  measuring  success  showed  outcomes  as  more  important  than 
processes  for  both  groups  and  corroborated  the  Likert  scale  mean  findings. 
Participants  who  work  on  equipment-related  services  rated  cost  as  more  important 
and  performance  as  less  important  on  both  Likert  and  ranking  questions  compared 
to  knowledge-based  participants.  This  finding  seems  to  indicate  that  for  knowledge- 
based  services,  stakeholders  were  more  willing  to  trade  cost  for  enhanced 
performance.  This  may  be  attributable  to  the  highly  specialized  and  technical  nature 
of  functions  such  as  engineering  management  and  program  management,  which  the 
government  lacks  the  crucial  internal  capacity  to  perform. 
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V. 


Summary,  Conclusions,  and 
Recommendations 


A.  Summary 

The  DoD’s  obligations  on  contracts  have  more  than  doubled  between  fiscal 
years  2001  and  2008  to  over  $387  billion,  with  over  $200  billion  spent  just  for 
services  in  2008  (GAO,  2009).  In  conjunction  with  this  increase  in  defense 
procurement  is  the  reduction  of  the  defense  acquisition  workforce.  The  combination 
of  the  increasing  defense  procurement  workload  and  the  decreasing  size  of  the 
government  workforce,  along  with  the  complexities  of  an  arcane  and  convoluted 
government  contracting  process,  have  created  the  perfect  storm — an  environment  in 
which  complying  with  government  contracting  policies  and  adopting  contract 
management  best  practices  has  not  always  been  feasible  (Rendon,  2010).  The 
contract  management  process  is  performed  with  inputs  from  the  different  functional 
areas,  using  a  cross-functional  team  or  integrated  project  team  (IPT)  structure.  The 
team  members  represent  their  functional  areas  such  as  program  management, 
contracting,  financial,  logistics,  and  quality  assurance.  Each  of  these  project  team 
members  represents  the  stakeholders,  and  their  different  goals  and  objectives.  The 
first  research  question  we  investigated  was  as  follows:  How  do  different 
stakeholders  define  successful  services  contracts  within  the  Navy?  To  develop  a 
clear  understanding  of  current  services  acquisition  practices,  we  also  investigated  a 
second  research  question:  How  do  different  stakeholders  measure  services 
contracts  within  the  Navy?  Investigating  the  above  two  questions  helped  us  develop 
recommendations  regarding  the  third  and  final  research  question:  How  should  the 
service  contract’s  success  be  measured? 

B.  Conclusions 

On  the  aggregate  level,  our  research  indicated  that,  when  defining  a 
successful  service  contract,  stakeholders  considered  outcomes  (in  the  order  of 
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performance,  cost,  and  schedule)  slightly  more  important  than  processes. 
Stakeholders  also  ranked  outcome-related  factors  as  most  important.  On  the 
aggregate,  our  research  indicated  that,  when  measuring  a  successful  service 
contract,  stakeholders  considered  outcomes  (in  the  order  of  cost,  schedule,  and 
performance)  more  important  than  processes.  Stakeholders  also  ranked  outcome- 
related  factors  as  most  important. 

On  the  stakeholder  level,  our  research  indicated  that,  when  defining  a 
successful  service  contract,  PMs,  CORs,  and  COs  considered  outcomes  (in  the 
order  of  performance,  cost,  and  schedule)  slightly  more  important  than  processes. 
PMs,  CORs,  and  COs  also  ranked  outcome-related  factors  as  most  important.  On 
the  stakeholder  level,  our  research  indicated  that,  when  measuring  a  successful 
service  contract,  PMs,  CORs,  and  COs  considered  outcomes  (in  the  order  of 
performance,  schedule,  and  cost)  more  important  than  processes.  PMs,  CORs,  and 
COs  also  ranked  outcome-related  factors  as  most  important. 

On  the  equipment-related  service  type,  our  research  indicated  that,  when 
defining  a  successful  service  contract,  stakeholders  considered  outcomes  (in  the 
order  of  performance,  schedule,  and  cost)  slightly  more  important  than  processes. 
Stakeholders  also  ranked  outcome-related  factors  as  most  important.  On  the 
equipment-related  service  type,  our  research  indicated  that,  when  measuring  a 
successful  service  contract,  stakeholders  considered  outcomes  (in  the  order  of  cost, 
performance,  and  schedule)  more  important  than  processes.  Stakeholders  also 
ranked  outcome-related  factors  as  most  important. 

On  the  knowledge-based  service  type,  our  research  indicated  that,  when 
defining  a  successful  service  contract,  stakeholders  considered  outcomes  (in  the 
order  of  performance,  schedule,  and  cost)  slightly  more  important  than  processes. 
Stakeholders  also  ranked  outcome-related  factors  as  most  important.  On  the 
knowledge-based  service  type,  our  research  indicated  that,  when  measuring  a 
successful  service  contract,  stakeholders  considered  outcomes  (in  the  order  of 
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performance,  cost,  and  schedule)  more  important  than  processes.  Stakeholders 
also  ranked  outcome-related  factors  as  most  important. 

C.  Recommendations 

Our  research  findings  have  several  implications  for  the  Navy,  as  well  as  the 
DoD.  All  stakeholders  surveyed  identified  and  ranked  outcome-related  factors  as 
more  important  than  process-related  factors,  in  both  defining  and  measuring  the 
success  of  service  contracts.  This  may  be  because  outcome-related  factors  (cost, 
schedule,  and  performance)  are  more  easily  defined  and  measured  using  available 
metrics,  compared  to  contracting  processes,  which  are  more  difficult  to  define  and 
many  agencies  have  no  available  metrics.  However,  as  discussed  in  the  earlier 
sections  of  this  paper,  many  of  the  contracting  deficiencies  identified  by  the  GAO 
and  DoDIG  are  related  to  contracting  processes,  such  as  conducting  market 
research,  determining  item  commerciality,  selecting  contract  type,  negotiating  fair 
and  reasonable  prices,  and  monitoring  contractors  through  surveillance.  Thus,  our 
first  recommendation  is  that  the  U.S.  Navy  should  develop  and  implement  process- 
related  metrics  to  define  and  measure  critical  contracting  processes,  such  as 
conducting  market  research,  determining  item  commerciality,  selecting  contract  type, 
negotiating  fair  and  reasonable  prices,  and  monitoring  contractors. 

Our  literature  review  identified  that  acquisition  stakeholders  (PMs,  CORs,  and 
COs)  have  different  procurement  goals  and  objectives,  and  these  goals  and 
objectives  may  in  fact  conflict  with  each  other.  Our  second  recommendation  is  that 
the  U.S.  Navy  should  establish  internal  controls  to  ensure  the  contracting  processes 
are  being  followed  and  that  the  different  stakeholders  place  sufficient  importance  on 
the  value  of  these  contracting  processes. 

Finally,  as  previous  research  has  determined  that  contracts  are  only  as 
successful  as  the  processes  used  to  plan,  award,  and  administer  these  contracts, 
our  final  recommendation  is  for  the  U.S.  Navy  to  implement  a  program  for 
continuously  assessing  its  contracting  process  capability  and  using  the  assessment 
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results  to  improve  its  organizational  contract  management  process  capability.  Once 
the  U.S.  Navy,  as  well  as  the  DoD,  implement  contracting  process-related  metrics  to 
define  and  measure  services  contracts,  internal  controls  to  ensure  contracting 
process  compliance,  and  periodical  assessments  of  organizational  contracting 
process  capability,  the  importance  of  process-related  factors  in  defining  and 
measuring  the  success  of  service  contracts  will  increase  among  stakeholders  and 
thus  start  addressing  some  of  the  contracting  deficiencies  identified  by  the  GAO  and 
the  DoDIG. 
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Appendix  A.  Tables  of  Survey  Results 


Table  2.  Aggregate  Stakeholders  Means  With  Distributions 


Aggregate 

Stakeholders 

Define 

Measure 

Likert 

Mean 

1 

2 

3 

4 

5 

Likert 

Mean 

1 

2 

3 

4 

5 

Process 

3.97 

31 

10 

3 

14 

7 

34 

8 

41 

9 

2.47 

85 

83 

57 

44 

26 

Outcome 

4.08 

9 

58 

16 

1 

53 

7 

43 

7 

3.80 

46 

11 

9 

21 

9 

36 

3 

19 

8 

Cost 

4.03 

3 

19 

49 

15 

8 

10 

6 

4.04 

0 

3 

27 

90 

39 

Schedule 

3.93 

2 

29 

56 

16 

4 

96 

4.00 

1 

5 

23 

96 

36 

Performance 

4.29 

4 

10 

56 

21 

5 

23 

5 

3.36 

45 

11 

1 

16 

9 

17 

7 

12 

3 

Table  3.  Aggregate  Stakeholders  Highest  Ranking  % 


Aggregate 

Stakeholders 

Define 

Measure 

Highest  Rank  % 

Highest  Rank  % 

Process 

40% 

46% 

Outcome 

60% 

54% 

Cost 

15% 

19% 

Schedule 

19% 

12% 

Performance 

26% 

24% 
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Table  4.  Stakeholder  Means  With  Distributions 


Stakeholders 

Define 

Measure 

Likert 

Mean 

1 

2 

3 

4 

5 

Likert 

Mean 

1 

2 

3 

4 

5 

PM 

Process 

3.87 

1 

11 

11 

30 

30 

2.52 

4 

8 

8 

4 

1 

Outcome 

4.13 

0 

4 

10 

40 

39 

3.78 

8 

8 

21 

25 

18 

Cost 

4.24 

0 

0 

3 

15 

9 

4.15 

0 

0 

1 

10 

3 

Schedule 

3.63 

0 

4 

6 

9 

7 

3.92 

0 

1 

2 

8 

3 

Performance 

4.51 

0 

0 

1 

16 

23 

3.25 

8 

7 

18 

7 

12 

COR 

Process 

3.97 

8 

12 

19 

55 

62 

2.76 

6 

15 

7 

11 

3 

Outcome 

4.15 

0 

9 

15 

52 

68 

3.77 

10 

22 

39 

49 

23 

Cost 

4.16 

0 

3 

5 

23 

19 

4.23 

0 

0 

3 

11 

8 

Schedule 

4.06 

0 

4 

8 

21 

19 

4.08 

0 

1 

2 

16 

6 

Performance 

4.23 

0 

2 

2 

8 

30 

2.99 

10 

21 

34 

22 

9 

CO 

Process 

3.97 

22 

75 

112 

239 

307 

2.41 

71 

56 

40 

28 

21 

Outcome 

4.06 

7 

43 

122 

390 

305 

3.81 

28 

82 

149 

271 

153 

Cost 

3.98 

3 

16 

40 

106 

76 

3.98 

0 

3 

23 

63 

27 

Schedule 

3.91 

2 

20 

39 

127 

63 

3.98 

1 

3 

19 

67 

26 

Performance 

4.27 

2 

7 

43 

157 

166 

3.47 

27 

76 

107 

141 

100 
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Table  5.  Stakeholders  Highest  Ranking  % 


Stakeholders 

Define 

Measure 

Highest  Rank  % 

Highest  Rank  % 

PM 

Process 

41% 

43% 

Outcome 

59% 

57% 

Cost 

11% 

13% 

Schedule 

22% 

14% 

Performance 

27% 

30% 

COR 

Process 

39% 

39% 

Outcome 

61% 

61% 

Cost 

17% 

28% 

Schedule 

19% 

9% 

Performance 

24% 

24% 

CO 

Process 

40% 

49% 

Outcome 

60% 

51% 

Cost 

15% 

17% 

Schedule 

17% 

12% 

Performance 

28% 

23% 
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Table  6.  Service  Type  Means  With  Distributions 


Service 

Type 

Define 

Measure 

Likert 

Mean 

1 

2 

3 

4 

5 

Likert 

Mean 

1 

2 

3 

4 

5 

Equipment  Related 

Process 

4.05 

7 

25 

69 

147 

159 

2.68 

27 

25 

23 

17 

14 

Outcome 

4.10 

2 

13 

70 

221 

160 

3.76 

17 

44 

94 

143 

72 

Cost 

4.03 

1 

5 

21 

64 

39 

4.08 

0 

0 

12 

39 

17 

Schedule 

3.97 

0 

6 

25 

70 

33 

3.87 

0 

2 

14 

39 

10 

Performance 

4.28 

1 

2 

24 

87 

88 

3.34 

17 

42 

68 

65 

45 

Knowledge  Based 

Process 

3.92 

15 

59 

52 

138 

179 

2.29 

43 

37 

20 

19 

7 

Outcome 

4.02 

6 

38 

60 

223 

180 

3.82 

20 

46 

88 

155 

88 

Cost 

4.00 

1 

11 

19 

65 

45 

3.97 

0 

2 

12 

39 

14 

Schedule 

3.82 

2 

21 

20 

63 

41 

4.07 

1 

2 

8 

37 

20 

Performance 

4.24 

3 

6 

21 

95 

94 

3.41 

19 

42 

68 

79 

54 

Table  7.  Service  Type  Highest  Rank  % 


Service  Type 

Define 

Measure 

Highest  Rank  % 

Highest  Rank  % 

Equipment  Related 

Process 

42% 

44% 

Outcome 

58% 

56% 

Cost 

15% 

21% 

Schedule 

18% 

13% 

Performance 

25% 

22% 

Knowledge  Based 

Process 

39% 

48% 

Outcome 

61% 

52% 

Cost 

16% 

16% 

Schedule 

18% 

11% 

Performance 

28% 

25% 

NPS 
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